In 2012, the World Wide Web Foundation launched for the first time the Web Index (WI), which combines the existing secondary data with new primary data to rank countries according to their progress and use of the Web. Primary data are gathered via a multi-country specifically designed questionnaire. The aim of our analysis is (1) to evaluate the measurement properties of the expert assessment survey and to provide survey designers with some insights into possible problematic questions and/or unexpectedly behaving countries and (2) to assess the experts’ perception of the state and the value of the Web. To do so the Rating Scale Rasch model is employed. Results show that about 10% of survey questions are detected as misfitting and need to be reworded. Possible reasons are: counter-orientation with respect to the WI polarity, difficulty in understanding the question's words or binary instead of the multiple response scale. Country analysis shows that no country can be considered as an outlier due to notable unexpected pattern of answers. Since the survey is to be expanded in future editions of the WI, the results of our analysis are very important in pin-pointing the questions most in need of refinement for the next edition of the Index.